Measuring Your Heart Rate Using Your Phone’s Camera and Flutter
Table of Contents
In this article, I’ll explain how you can develop a simple app with Flutter that measures heart rate variability and displays it in a chart using only the phone’s camera and flash.
. . .
Concept #
You’ve probably seen or know of devices that people clip to their fingers in hospitals that measure their heart rate, or smartwatches capable of measuring your heart rate. They all have one thing in common: They measure the heart rate with a technique called photoplethysmography.
A photoplethysmogram (PPG) is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue. — Wikipedia #
Shining a light into a blood irrigated tissue, we can measure the variability of reflected light and extract the variation of blood flow. As we all know, the blood flow is dependent on the heart rate, so we can calculate the heart rate using the blood flow variation.
So, in our application, we’ll shine the camera’s flash and measure the intensity reflected using the phone’s camera. More specifically, we’ll measure the average value of all the pixel’s intensity of the camera image. Then, if we cover the camera and flash with our finger, the intensity measured will vary with the blood flow.
. . .
Code #
Dependencies #
First, we need to install the dependencies:
- charts_flutter — Material Design data visualization library written natively in Dart.
- wakelock — This Flutter plugin allows you to enable and toggle the screen wakelock on Android and iOS, which prevents the screen from turning off automatically.
- camera — A Flutter plugin for iOS and Android allowing access to the device cameras.
...
dependencies:
flutter:
sdk: flutter
cupertino_icons: ^0.1.2
charts_flutter: ^0.9.0
wakelock: ^0.1.4+1
camera: ^0.7.0+4
...
Application #
Our application’s interface is divided into three files: main.dart, homePage.dart, and chart.dart.
main.dart - Here we only need to set the HomePage widget as our home widget, so it displays when the application runs:
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'homePage.dart';
void main() => runApp(MyApp());
class MyApp extends StatelessWidget {
@override
Widget build(BuildContext context) {
return MaterialApp(
title: 'PPG',
theme: ThemeData(
brightness: Brightness.light,
),
home: HomePage(),
);
}
}
HomePage - The core of the application is written on the homePage.dart file, which is our HomePage widget.
First, we need to create a Scaffold and, in its body, insert a centered IconButton, that will activate or deactivate the camera for the reading process.
import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:wakelock/wakelock.dart';
import 'chart.dart';
class HomePage extends StatefulWidget {
@override
HomePageView createState() {
return HomePageView();
}
}
class HomePageView extends State<HomePage> {
bool _toggled = false;
@override
Widget build(BuildContext context) {
return Scaffold(
backgroundColor: Colors.white,
body: SafeArea(
child: Center(
child: IconButton(
icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
color: Colors.red,
iconSize: 128,
onPressed: () {
if (_toggled) {
_untoggle();
} else {
_toggle();
}
},
),
),
),
);
}
}
As one can see, there is a boolean _toggled which stores the state of the IconButton. Now, we only need to define the functions _toggle and _untoggle.
For now, these functions will only change the value of the _toggle variable:
_toggle() {
setState(() {
_toggled = true;
});
}
_untoggle() {
setState(() {
_toggled = false;
});
}
The next step, is dividing the screen into three equal parts, using three Expanded widgets inside a Column:
@override
Widget build(BuildContext context) {
return Scaffold(
backgroundColor: Colors.white,
body: SafeArea(
child: Column(
children: <Widget>[
Expanded(
child: Container(
color: Colors.red,
),
),
Expanded(
child: Center(
child: IconButton(
icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
color: Colors.red,
iconSize: 128,
onPressed: () {
if (_toggled) {
_untoggle();
} else {
_toggle();
}
},
),
),
),
Expanded(
child: Container(
color: Colors.black,
),
),
],
),
),
);
}
On the bottom Container we display the real-time chart, where the camera’s data will be displayed. A margin and round corners were also added to this Container.
import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:wakelock/wakelock.dart';
import 'chart.dart';
class HomePage extends StatefulWidget {
@override
HomePageView createState() {
return HomePageView();
}
}
class HomePageView extends State<HomePage> {
bool _toggled = false;
List<SensorValue> _data = [];
...
@override
Widget build(BuildContext context) {
return Scaffold(
backgroundColor: Colors.white,
body: SafeArea(
child: Column(
children: <Widget>[
Expanded(
child: Container(
color: Colors.red,
),
),
Expanded(
child: Center(
child: IconButton(
icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
color: Colors.red,
iconSize: 128,
onPressed: () {
if (_toggled) {
_untoggle();
} else {
_toggle();
}
},
),
),
),
Expanded(
child: Container(
margin: EdgeInsets.all(12),
decoration: BoxDecoration(
borderRadius: BorderRadius.all(
Radius.circular(18),
),
color: Colors.black),
child: Chart(_data),
),
),
],
),
),
);
}
}
For the chart widget, we will use the package charts_flutter. The file chart.dart contains a StatelessWidget that displays in a chart points provided in a list. Each point is constituted by a DateTime value, which indicates the x value, and a double value that represents the y value.
import 'package:charts_flutter/flutter.dart' as charts;
import 'package:flutter/material.dart';
class Chart extends StatelessWidget {
final List<SensorValue> _data;
Chart(this._data);
@override
Widget build(BuildContext context) {
return new charts.TimeSeriesChart([
charts.Series<SensorValue, DateTime>(
id: 'Values',
colorFn: (_, __) => charts.MaterialPalette.green.shadeDefault,
domainFn: (SensorValue values, _) => values.time,
measureFn: (SensorValue values, _) => values.value,
data: _data,
)
],
animate: false,
primaryMeasureAxis: charts.NumericAxisSpec(
tickProviderSpec:
charts.BasicNumericTickProviderSpec(zeroBound: false),
renderSpec: charts.NoneRenderSpec(),
),
domainAxis: new charts.DateTimeAxisSpec(
renderSpec: new charts.NoneRenderSpec()));
}
}
class SensorValue {
final DateTime time;
final double value;
SensorValue(this.time, this.value);
}
Regarding the upper Container, we wish to divide it into two halves. On the left one, will display the CameraPreview and, on the right half a Text containing the Beats Per Minute (BPM).
class HomePageView extends State<HomePage> {
bool _toggled = false;
List<SensorValue> _data = [];
CameraController _controller;
...
@override
Widget build(BuildContext context) {
return Scaffold(
backgroundColor: Colors.white,
body: SafeArea(
child: Column(
children: <Widget>[
Expanded(
child: Row(
children: <Widget>[
Expanded(
child: Center(
child: _controller == null
? Container()
: CameraPreview(_controller),
),
),
Expanded(
child: Center(
child: Text("BPM"),
),
),
],
),
),
Expanded(
child: Center(
child: IconButton(
icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
color: Colors.red,
iconSize: 128,
onPressed: () {
if (_toggled) {
_untoggle();
} else {
_toggle();
}
},
),
),
),
Expanded(
child: Container(
margin: EdgeInsets.all(12),
decoration: BoxDecoration(
borderRadius: BorderRadius.all(
Radius.circular(18),
),
color: Colors.black),
child: Chart(_data),
),
),
],
),
),
);
}
}
The CameraPreview widget requires a valid CameraController. So we need to initialize the controller once we press the heart button and it’s toggled. Once the button is untoggled, we must also dispose of the controller:
Future<void> _initController() async {
try {
List _cameras = await availableCameras();
_controller = CameraController(_cameras.first, ResolutionPreset.low);
await _controller.initialize();
} catch (Exception) {
print(Exception);
}
}
_disposeController() {
_controller.dispose();
_controller = null;
}
Now, we only need to integrate these functions in the functions that toggle the button.
_toggle() {
_initController().then((onValue) {
setState(() {
_toggled = true;
});
});
}
_untoggle() {
_disposeController();
setState(() {
_toggled = false;
});
}
We should also override the dispose method and dispose of the CameraController once the application is disposed.
@override
void dispose() {
_disposeController();
super.dispose();
}
We must not forget to activate the camera’s flash and then start the ImageStream, which will provide the images that we will process. The function _scanImage will take care of it.
Future<void> _initController() async {
try {
List _cameras = await availableCameras();
_controller = CameraController(_cameras.first, ResolutionPreset.low);
await _controller.initialize();
Future.delayed(Duration(milliseconds: 100)).then((onValue) {
_controller.setFlashMode(FlashMode.torch);
});
_controller.startImageStream((CameraImage image) {
_scanImage(image);
});
} catch (Exception) {
print(Exception);
}
}
The function _scanImage calculates the average of the camera image’s red channel and adds the value to the data list, which is displayed on the chart explained above. Notice that we limit the number of points of the data list to 50 values.
_scanImage(CameraImage image) {
double _avg =
image.planes.first.bytes.reduce((value, element) => value + element) /
image.planes.first.bytes.length;
if (_data.length >= 50) {
_data.removeAt(0);
}
setState(() {
_data.add(SensorValue(DateTime.now(), _avg));
});
}
We do not need to process every frame, therefore, we can select a sampling rate. In this example, a sampling rate of 30 samples/seconds is used. For that purpose, we have a boolean, _processing, which becomes true once the _scanImage function is called and stays that way for 1/30 seconds, then returns to false. The _scanImage function will only be executed if the boolean _processing is false.
Future<void> _initController() async {
try {
List _cameras = await availableCameras();
_controller = CameraController(_cameras.first, ResolutionPreset.low);
await _controller.initialize();
Future.delayed(Duration(milliseconds: 100)).then((onValue) {
_controller.setFlashMode(FlashMode.torch);
});
_controller.startImageStream((CameraImage image) {
if (!_processing) {
setState(() {
_processing = true;
});
_scanImage(image);
}
});
} catch (Exception) {
print(Exception);
}
}
_scanImage(CameraImage image) {
double _avg =
image.planes.first.bytes.reduce((value, element) => value + element) /
image.planes.first.bytes.length;
if (_data.length >= 50) {
_data.removeAt(0);
}
setState(() {
_data.add(SensorValue(DateTime.now(), _avg));
});
Future.delayed(Duration(milliseconds: 1000 ~/ 30)).then((onValue) {
setState(() {
_processing = false;
});
});
}
For good practice, we should also set the _processing value to false every time we change the heart button’s toggled state.
_toggle() {
_initController().then((onValue) {
setState(() {
_toggled = true;
_processing = false;
});
});
}
_untoggle() {
_disposeController();
setState(() {
_toggled = false;
_processing = false;
});
}
We now have an application that measures the blood flow volume variability and displays it in a chart. Now we only need to calculate the heart rate, which is the frequency of the plotted signal. I used a simple algorithm that measures the average and the max along our window data, sets the threshold to the mean of those values, and detects the peaks above that threshold. It then updates the BPM value with an attenuation coefficient so we don’t have abrupt changes.
class HomePageView extends State<HomePage> {
...
double _alpha = 0.3;
_toggle() {
_initController().then((onValue) {
setState(() {
_toggled = true;
_processing = false;
});
_updateBPM();
});
}
...
_updateBPM() async {
List<SensorValue> _values;
double _avg;
int _n;
double _m;
double _threshold;
double _bpm;
int _counter;
int _previous;
while (_toggled) {
_values = List.from(_data);
_avg = 0;
_n = _values.length;
_m = 0;
_values.forEach((SensorValue value) {
_avg += value.value / _n;
if (value.value > _m) _m = value.value;
});
_threshold = (_m + _avg) / 2;
_bpm = 0;
_counter = 0;
_previous = 0;
for (int i = 1; i < _n; i++) {
if (_values[i - 1].value < _threshold &&
_values[i].value > _threshold) {
if (_previous != 0) {
_counter++;
_bpm +=
60000 / (_values[i].time.millisecondsSinceEpoch - _previous);
}
_previous = _values[i].time.millisecondsSinceEpoch;
}
}
if (_counter > 0) {
_bpm = _bpm / _counter;
setState(() {
_bpm = (1 - _alpha) * _bpm + _alpha * _bpm;
});
}
await Future.delayed(Duration(milliseconds: (1000 * 50 / 30).round()));
}
}
...
}
The last thing we need to do is avoid the screen from turning off while the :
_toggle() {
_initController().then((onValue) {
Wakelock.enable();
setState(() {
_toggled = true;
_processing = false;
});
_updateBPM();
});
}
_untoggle() {
_disposeController();
Wakelock.disable();
setState(() {
_toggled = false;
_processing = false;
});
}
At last, we have an application that measures the blood volume variability and estimates the BPM.
The full code:
import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:wakelock/wakelock.dart';
import 'chart.dart';
class HomePage extends StatefulWidget {
@override
HomePageView createState() {
return HomePageView();
}
}
class HomePageView extends State<HomePage> {
bool _toggled = false;
bool _processing = false;
List<SensorValue> _data = [];
CameraController _controller;
double _alpha = 0.3;
int _bpm = 0;
_toggle() {
_initController().then((onValue) {
Wakelock.enable();
setState(() {
_toggled = true;
_processing = false;
});
_updateBPM();
});
}
_untoggle() {
_disposeController();
Wakelock.disable();
setState(() {
_toggled = false;
_processing = false;
});
}
Future<void> _initController() async {
try {
List _cameras = await availableCameras();
_controller = CameraController(_cameras.first, ResolutionPreset.low);
await _controller.initialize();
Future.delayed(Duration(milliseconds: 100)).then((onValue) {
_controller.setFlashMode(FlashMode.torch);
});
_controller.startImageStream((CameraImage image) {
if (!_processing) {
setState(() {
_processing = true;
});
_scanImage(image);
}
});
} catch (Exception) {
print(Exception);
}
}
_updateBPM() async {
List<SensorValue> _values;
double _avg;
int _n;
double _m;
double _threshold;
double _bpm;
int _counter;
int _previous;
while (_toggled) {
_values = List.from(_data);
_avg = 0;
_n = _values.length;
_m = 0;
_values.forEach((SensorValue value) {
_avg += value.value / _n;
if (value.value > _m) _m = value.value;
});
_threshold = (_m + _avg) / 2;
_bpm = 0;
_counter = 0;
_previous = 0;
for (int i = 1; i < _n; i++) {
if (_values[i - 1].value < _threshold &&
_values[i].value > _threshold) {
if (_previous != 0) {
_counter++;
_bpm +=
60000 / (_values[i].time.millisecondsSinceEpoch - _previous);
}
_previous = _values[i].time.millisecondsSinceEpoch;
}
}
if (_counter > 0) {
_bpm = _bpm / _counter;
setState(() {
_bpm = (1 - _alpha) * _bpm + _alpha * _bpm;
});
}
await Future.delayed(Duration(milliseconds: (1000 * 50 / 30).round()));
}
}
_scanImage(CameraImage image) {
double _avg =
image.planes.first.bytes.reduce((value, element) => value + element) /
image.planes.first.bytes.length;
if (_data.length >= 50) {
_data.removeAt(0);
}
setState(() {
_data.add(SensorValue(DateTime.now(), _avg));
});
Future.delayed(Duration(milliseconds: 1000 ~/ 30)).then((onValue) {
setState(() {
_processing = false;
});
});
}
_disposeController() {
_controller.dispose();
_controller = null;
}
@override
void dispose() {
_disposeController();
super.dispose();
}
@override
Widget build(BuildContext context) {
return Scaffold(
backgroundColor: Colors.white,
body: SafeArea(
child: Column(
children: <Widget>[
Expanded(
child: Row(
children: <Widget>[
Expanded(
child: Center(
child: _controller == null
? Container()
: CameraPreview(_controller),
),
),
Expanded(
child: Center(
child: Text(
(_bpm > 30 && _bpm < 150 ? _bpm.round().toString() : "--"),
style: TextStyle(fontSize: 32, fontWeight: FontWeight.bold),
),
),
),
],
),
),
Expanded(
child: Center(
child: IconButton(
icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
color: Colors.red,
iconSize: 128,
onPressed: () {
if (_toggled) {
_untoggle();
} else {
_toggle();
}
},
),
),
),
Expanded(
child: Container(
margin: EdgeInsets.all(12),
decoration: BoxDecoration(
borderRadius: BorderRadius.all(
Radius.circular(18),
),
color: Colors.black),
child: Chart(_data),
),
),
],
),
),
);
}
}
Here’s the working application:
The GitHub repository for this project can be found here
On my GitHub repository, you’ll notice that I updated the application to using a Timer.periodic, which presented better results. I also added more customization and an animation the heart-shaped button that simulates a beating heart. I didn’t include it in this tutorial since its not necessary.